Smart load balancer and throttle

ABSTRACT

A load balancing system for a packet switched communications network, the load balancing system comprising: at least one stateful load balancer configured to receive and process copies of control plane packets propagated by the network to determine a characterizing vector for each target feature of at least one target feature that characterizes packets propagated by the network, which vector comprises one or more characterizing components and is useable to associate packets propagated by the network with the target feature; and at least one non-stateful load balancer configured to: receive the at least one characterizing vector and copies of packets propagated by the network; use the at least one characterizing vector to associate a target feature of the at least one target feature with each of a plurality of the received packets; and load balance and transmit copies of packets that the non-stateful load balancer associates with a target feature of the at least one target feature to at least one probe, and copies of packets associated with a same target feature to a same probe of the at least one probe.

RELATED APPLICATIONS

The present application is a continuation in part of U.S. patentapplication Ser. No. 15/325,462 filed on Jan. 11, 2017, now U.S. Pat.No. 10,050,886 issued on Aug. 14, 2018, which is a U.S. National Phaseof PCT Application PCT/IB2016/054332, filed on Jul. 21, 2016, whichclaims the benefit under 35 U.S.C. 119 (e) of U.S. ProvisionalApplication 62/334,170 filed on May 10, 2016, the disclosures of whichare incorporated herein by reference.

FIELD

Embodiments of the disclosure relate to load balancing tasks comprisedin monitoring a communication network

BACKGROUND

Modern communication networks support large, varied, and growingpopulations of users and an ever increasing gamut of user services anduser applications for users who access the networks using any of variousstationary and/or mobile user equipment (UE). The networks span not onlyglobal populations and geography, but a plethora of methods, devices,infrastructures, and protocols for propagating data, voice, and videocontent provided by a host of different large and small serviceproviders.

As the various software and hardware resources that support thenetworks, configure paths along which network packets propagate in thenetworks, and provide services mediated by the networks have become moresophisticated, the resources, which may be referred to as networkequipment (NE), have become fungible and logically abstracted away fromtheir particular physical structures. The physical devices and apparatusunderlying modern communication networks have become commodity resourcesthat are configurable by software to provide virtualizations of networkfunctions, referred to as “Network Functions Virtualization” (NFV), and“Software Defined Networks” (SDNs). And dedicated physical devices thatprovide particular network functionalities have been or are beingreplaced by software entities that access the physical commodities of anetwork, and/or other software entities of the network, on an “as need”basis to communicate with each other and provide functionalitiesrequired by the networks. The zoology of software entities areconventionally referred to as virtual entities, with a particularvirtual entity typically distinguished by a name of a functionality itprovides.

The extensive, complex infrastructure of many modern communicationnetworks typically comprise a mix of dedicated physical devices and anever increasing, if not dominant number, of virtual network entitiesengaged in an incessant packet exchange chatter approaching hundreds ofgigabits per second (Gbps). Monitoring and troubleshooting the networksas much as possible in real time to maintain performance that providesacceptable quality of service (QoS) and user quality of experience (QoE)for the multitude of services that the networks provide has become ademanding challenge.

SUMMARY

An aspect of an embodiment of the disclosure relates to providing anoptionally virtualized load balancing system that is relatively easilyscalable and configured to efficiently distribute data propagating in apacket switched communication network to probes and/or other networkmonitoring entities, such as network security devices, networkanalyzers, and other network monitoring tools, generically referred toas probes, that operate to process the data and provide the network withinformation relevant to understanding, monetizing, and/or controlling,optionally, network performance. In an embodiment, the virtualized loadbalancing system, which may be referred to as a “Smart-vLB system” orsimply a “Smart-vLB”, comprises a hierarchical architecture comprisingat least one “non-stateful” virtual load balancer (NS-vLB) and at leastone stateful virtual load balancer (S-vLB). The at least one NS-vLB andat least one S-vLB cooperate to distribute data in packets that theSmart-vLB receives from the network for processing, to a plurality ofprobes, optionally comprising virtual probes (vProbes). In anembodiment, the communication network comprises an evolved packet system(EPS) network, which as discussed below may also be referred to as anLTE (long term evolution) network.

The at least one NS-vLB in the Smart-vLB receives packets, which may becontrol plane and/or user plane packets, that may be mirrored from atleast one port in the network and distinguishes between packets,optionally referred to as “contextual packets”, that it receives, andpackets, optionally referred to as “non-contextual packets”, which itreceives. A contextual packet is a packet comprising data that may notbe processed for a given desired purpose without reference to data fromanother, associated contextual packet and may advantageously beprocessed by a same virtual probe that processes data from theassociated contextual packet. For example, a contextual packet maycomprise data that may not be directly associated with a session ortransaction to which it belongs but may be associated with another,associated, packet to determine a session or transaction to which thecontextual packet belongs. Packets may be deemed associated by sharing acommon attribute, such as by way of example, sharing a same networkprotocol or application, and/or belonging to a same network session ortransaction. A non-contextual packet, is a packet comprising data thatmay be processed for a given desired purpose without reference to datafrom another packet and may for the given desired purpose be processedby a probe that is different from a probe that processes other packetsfor the same desired purpose. A non-contextual packet may by way ofexample be a packet that may be directly associated with a session ortransaction to which it belongs without reference to data in anotherpacket.

In an embodiment, each of the at least one NS-vLB is configured totransmit non-contextual packets that it receives for processing to agiven physical or virtual probe of optionally a plurality of physical orvirtual probes without reference to information in another packet. TheNS-vLB is configured to transmit associated contextual packetscharacterized by a same shared attribute to a same S-vLB. The S-vLBtransmits contextual packets for processing to a vProbe having anidentity which the S-vLB determines based on information from at leastone of a plurality of other packets that it receives. The S-vLBtransmits a contextual packet that it receives for processing to avProbe having an identity which the S-vLB determines based oninformation from at least one other packet that the S-v-LB receives.

In an embodiment the Smart-vLB is configured to load balance packets andcontrol, “throttle”, transmission rates of copies of packets to a probefor processing to prevent oversubscribing probe capacity. Optionally,load balancing and throttling transmission rates comprise associatingthe packets with a particular feature, also referred to as a targetfeature, that characterizes the packets and load balancing andthrottling based on the particular feature. In an embodiment the targetfeature of the packets may by way of example, be an IMSI or an IMEI(International Mobile Equipment Identity) of a UE associated withtransmission of the packets, an MSISDN (Mobile Station InternationalSubscriber Directory Number), APN (Access Point Name), QCI, (Quality ofService Class Identifier), and/or a CGI (Cell Global Identifier).

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF FIGURES

Non-limiting examples of embodiments of the disclosure are describedbelow with reference to figures attached hereto that are listedfollowing this paragraph. Identical features that appear in more thanone figure are generally labeled with a same label in all the figures inwhich they appear. A label labeling an icon representing a given featureof an embodiment of the disclosure in a figure may be used to referencethe given feature. Dimensions of features shown in the figures arechosen for convenience and clarity of presentation and are notnecessarily shown to scale.

FIG. 1 schematically shows a long term evolution, LTE, communicationsnetwork;

FIG. 2 schematically shows a Smart-vLB comprising NS-vLBs and S-vLBsoperating to load balance processing of data comprised in controlpackets transmitted by network entities of the LTE shown in FIG. 1, inaccordance with an embodiment of the disclosure;

FIG. 3 shows a flow diagram of a procedure by which the NS-vLBscomprised in the Smart-vLB shown in FIG. 2 distribute packets that theSmart-vLB receives from the LTE network, in accordance with anembodiment of the disclosure;

FIGS. 4A and 4B show a flow diagram of a procedure by which one of theS-vLBs comprised in the Smart-vLB shown in FIG. 2 distribute packetsthat it receives from a NS-vLB in the Smart-vLB, in accordance with anembodiment of the disclosure; and

FIGS. 5A-5C show a flow diagram of a procedure by which another of theS-vLBs comprised in the Smart-vLB shown in FIG. 2 distribute packetsthat it receives from a NS-vLB in the Smart-vLB, in accordance with anembodiment of the disclosure;

FIG. 6 schematically shows a Smart-vLB comprising NS-vLBs and S-vLBsoperating to load balance processing of data comprised in controlpackets propagated to and from a Serving Gateway (SGW), and/or a PacketData Network Gateway (PGW) of the LTE shown in FIG. 1, in accordancewith an embodiment of the disclosure;

FIGS. 7A and 7B show flow diagrams of procedures by which and NS-vLB andan S-vLB comprised in the Smart-vLB shown in FIG. 6 operate todistribute packets, in accordance with an embodiment of the disclosure;and

FIG. 8 shows a flow diagram of a procedure by which the Smart-vLB shownin FIG. 6 may operate to load balance and throttle transmission rates ofpackets transmitted to probes for processing, in accordance with anembodiment of the disclosure.

DETAILED DESCRIPTION

In the following detailed description, a Smart-vLB that is configured inaccordance with an embodiment of the disclosure to load balanceprocessing of control data transmitted in packets by entities comprisedin an LTE communications network is described with reference to FIGS.1-7B. Features and functioning of the LTE network are discussed withreference to FIG. 1. FIG. 2 schematically shows the Smart-vLB operatingto load balance processing of data in control packets transmitted toand/or from an MME comprised in the LTE network shown in FIG. 1. Whereasthe Smart-vLB in FIG. 2 is shown by way of example comprising twoNS-vLBs and two S-vLBs, practice of an embodiment of the disclosure isnot limited to Smart-vLBs comprising two each of NS-vLBs and S-vLBs. ASmart-vLB in accordance with an embodiment of the disclosure maycomprise more or less than two NS-vLBs and more or less than two S-vLBs.Procedures by which component NS-vLBs and S-vLBs in the Smart-vLBprocess packets that the Smart-vLB receives from the LTE network arediscussed with reference to the flow diagrams shown in FIGS. 3A-5C,7A-7B, and 8.

In the discussion, unless otherwise stated, adjectives such as“substantially” and “about” modifying a condition or relationshipcharacteristic of a feature or features of an embodiment of thedisclosure, are understood to mean that the condition or characteristicis defined to within tolerances that are acceptable for operation of theembodiment for an application for which the embodiment is intended.Wherever a general term in the disclosure is illustrated by reference toan example instance or a list of example instances, the instance orinstances referred to, are by way of non-limiting example instances ofthe general term, and the general term is not intended to be limited tothe specific example instance or instances referred to. Unless otherwiseindicated, the word “or” in the description and claims is considered tobe the inclusive “or” rather than the exclusive or, and indicates atleast one of, or any combination of more than one of items it conjoins.

FIG. 1 schematically shows network elements comprised in a mobilecommunication network 18 comprising a 3rd Generation Wireless MobileCommunication (3G) network 19 and an evolved packet system (EPS)communications network 20 that operates to connect a user's mobilecommunication equipment, (UE) to one or more Packet Data Networks(PDNs), and/or, as a circuit switch fallback option, to the PublicSwitched Telephone Network (PSTN). Network 20 may operate to provide auser's UE with communications to the Internet and/or to stationaryand/or mobile communication equipment of other users (not shown). InFIG. 1 mobile user equipment, UE, is schematically represented bysmartphone and laptop icons 22.

EPS network 20 comprises a packet switched core network referred to asan Evolved Packet Core (EPC) core network indicated by a dashed boundarylabeled EPC and a Long Term Evolution (LTE) radio access networkreferred to as an Evolved Universal Terrestrial Radio Access NetworkE-UTRAN indicated by a dashed boundary labeled E-UTRAN. E-UTRAN providesuser equipment with wireless radio frequency (RF) channel connectivityto the core network EPC. The core network EPC provides a UE connected tothe core network via an E-UTRAN radio channel with a communicationsconnection to a PDN via an internet protocol (IP) packet flow“pipeline”, conventionally referred to as a “bearer”. The acronym LTE isconventionally used when referring to an evolved packet system (EPS)network comprising both the LTE E-UTRAN and the evolved packet core EPC,and network 20 shown in FIG. 1 may be referred to as LTE network 20, orsimply LTE.

E-UTRAN comprises a network of RF communication base stations referredto as evolved NodeB (eNodeB) base stations. Each eNodeB comprises an RFtransceiver and integral controller that controls the RF transceiver toestablish radio communication channels between UEs in a limitedgeographic region referred to as a cell, and the core network EPC.Legacy base stations, referred to as NodeB base stations used in 3Gnetworks, do not comprise integral controllers but are connected toradio network controllers (RNCs), operate using RF transmissiontechnologies different from those used by eNodeBs and have limitedfunctionalities in comparison to eNodeB base stations. Core network EPCis capable of communicating with and connecting a UE connected to aNodeB with a PDN network, and LTE network 20 is schematically shownconnected to a NodeB base station of 3G network 19 via an RNC and aServing GPRS (General Packet Radio Service) Support Node, (SGSN) of the3G network. The SGSN node is responsible for delivery of packets fromand to UEs that are connected to a NodeB base station rather than aneNodeB base station. The SGSN is connected to the Internet via a GGSN(GPRS Gateway Support Node). Whereas in FIG. 1 mobile network 18 isshown comprising a E-UTRAN having two eNodeBs and a 3G network havingone NodeB, an LTE may and typically does comprise more than two eNodeBsand a 3G network comprising more than one NodeB.

Core network EPC of LTE 20 comprises a plurality of logical nodes, whichare supported by virtual and/or physical network entities. The networknodes include, a Serving Gateway (SGW), a PDN gateway (PGW), a HomeSubscriber Server (HSS), a Mobile Switching Center (MSC) server, and aMobile Management Entity (MME). The SGW is a node of the LTE networkthrough which data packets from a UE connected to an eNodeB enter theEPC to be propagated from the UE through the EPC to their destinations,or exit the EPC to propagate to the eNodeB and the UE after propagatingthrough the EPC from their sources. The SGW serves as a local mobilityanchor for data bearers when the UE moves between cells of differenteNodeBs. The PGW is a node through which UE data packets from a UEconnected to the EPC exit the EPC to propagate to a PDN or enter the EPCfrom a PDN to propagate to the UE. The PGW is responsible for allocatingan IP address to the UE and handling various administrative and chargingtasks in accordance with information received by the PGW from a nodereferred to as a Policy and Charging Rules Function (PCRF) node. The HSSnode contains user subscription data, such as a QoS profile and anyrestriction that might limit user roaming. The HSS also contains dynamicinformation such as an identity of an MME to which the user UE maycurrently be connected. The MSC node supports signaling between an MMEand a UE Subscriber Identity Module (SIM) and Short Message Services(SMS) for a UE connected to the EPC via a NodeB.

The MME is a central control and signaling node that communicates withand orchestrates the activities of the other nodes and the UE inmanaging the resources of the LTE to establish, route, and maintain IPpacket flow between a UE and a PDN. An EPC generally comprises more thanone MME and may hand over connection to a UE from one to another of theMMEs as the UE moves between cells of different eNodeBs. In FIG. 1 LTE20 is shown comprising two MMEs, a first MME 31 represented by a solidline rectangle and indicated by dotted lines as communicating with UEsshown in the figures, and a second MME 32, shown in dashed lines, towhich first MME 31 may hand over UEs as the UEs move between cells.

Communication between two nodes in an LTE network is governed by aninterface that configures what data the nodes exchange, how packets thatcontain the data are formatted, and how the packets are transmittedbetween the nodes. Communication between different node pairs istypically governed by different interfaces. An interface that configurescommunications between two nodes is indicated in FIG. 1 by analphanumeric identifying the interface shown in an ellipse overlying adashed or solid line that extends between the two nodes. For example,eNodeBs in an LTE network exchange control signaling via an X2interface, and an X2 is shown in an ellipse overlying a dashed lineshown connecting the two eNodeBs in LTE network 20, and a UEcommunicates with an eNodeBs via a Uu interface shown in an ellipseoverlying a solid line between the UE and the eNode B. Similarly asindicated in FIG. 1, the MME node communicates control data with aneNodeB via an S1-MME interface. The S1-MME interface defines data to beexchanged and packet formatting in accordance with an S1-AP protocol andconfigures transmission of the data between the MME and the eNodeB usingthe SCTP (Stream Control Transmission Protocol) transport protocol. TheMME node communicates with the SGW node via an S11 interface, whichtunnels UDP packets between the nodes using a General Packet RadioService Tunneling Protocol (GTP) referred to as GTP-C. And, the HSScommunicates with the MME via an S6a interface that configures data inaccordance with a Diameter protocol and transmits Diameter messagesusing SCTP.

FIG. 2 schematically shows a Smart-vLB 100 coupled to LTE network 20operating to load balance processing of data comprised in controlpackets transmitted between MME node 31 and other nodes in the LTEnetwork, in accordance with an embodiment of the disclosure. By way ofexample, Smart-vLB 100 is assumed to comprise two, optionally virtual,non-Stateful load balancers, NS-vLBs, 101 and 102, and two, optionallyvirtual, Stateful load balancers, S-vLB 110 and S-v1B 111, and is shownload balancing packets it receives from LTE network 20 between,optionally three, vProbes 121, 122, and 123.

It is noted that whereas Smart-vLB 100 is schematically shown in FIG. 2and assumed to comprise virtual component entities, any or anycombination of component entities of a Smart-vLB in accordance with anembodiment, such a non-Stateful load balancer and/or a Stateful loadbalancer may comprise and/or have access to any of various physicaland/or virtual processors and memories that may be required to supportfunctionalities for which the component entities are responsible. Acomponent entity may be a physical or virtual entity or a combination ofphysical and virtual entities and may comprise or have access to amemory having any electronic and/or optical circuitry suitable forstoring data and/or computer executable instructions. The memory may, byway of example, comprise or have access to any one, or any combinationof more than one of a flash memory, random access memory (RAM), readonly memory (ROM), and/or erasable programmable read-only memory(EPROM). Similarly, a component entity may comprise and/or have accessto a processor having any electronic and/or optical processing circuitryconfigured to operate and provide functionalities that the componententity may require. The processor may by way of example comprise anyone, or any combination of more than one of, a standard x86 processor,microprocessor, an application specific circuit (ASIC), fieldprogrammable array (FPGA) and/or system on a chip (SOC).

Smart-vLB 100 is shown in FIG. 2 by way of example coupled to LTE 20 toreceive control packets mirrored from interfaces S1-MME, S3, and Gn,S11, Sv and SGs, S6a and S10 between MME 31, and eNodeBs, SGSN, SGW,MSC, HSS, and MME32, respectively. Link controllers (not shown) in MME31 are configured to distribute substantially equal numbers of mirroredpackets from LTE network 20 to each non-Stateful vLB, NS-vLB 101 and102. Diameter packets mirrored to Smart-vLB 100 from interface S6a andGTP-C tunneled packets mirrored from interfaces, Sv, S3, S11 and S10 areselected as contextual packets. Packets mirrored to Smart-vLB 100 frominterfaces S1-MME and SGs are considered non-contextual packets. EachNS-vLB 101 and 102 forwards non-contextual packets that it receives insubstantially equal numbers to a given vProbe 121, 122, and 123 withoutreference to data in another packet. On the other hand, both NS-vLB 101and 102 forward Diameter packets that they receive to S-vLB 111, andGTP-C packets they receive to S-vLB 110. Each S-vLB load balancerforwards a packet it receives to a vProbe 121, 122, or 123 having anidentity which the S-vLB determines based on information from at leastone of a plurality of other packets that it receives.

For convenience of reference, S-vLBs 110 and 111 are labeled by thetypes of packets they respectively receive. S-vLB 110 is labeled “GTP-C”and S-vLB 111 is labeled “Diameter”.

FIG. 3 shows a flow diagram of an algorithm 200 by which NS-vLB 101 orNS-vLB 102, generically referred to as an NS-vLB, may determine where toforward packets they receive, in accordance with an embodiment of thedisclosure.

In a block 202 the NS-vLB receives and optionally time stamps a packetfrom the selected MME interfaces shown in FIG. 2, and in a decisionblock 204 determines if the packet contains Application, Layer 7 of theOpen System Interconnection (OSI) model, data. If the packet does notcontain L7 data, in a block 205 the NS-vLB discards the packet andreturns to block 202. If on the other hand it is an L7 packet, NS-vLBmay proceed to a block 206 to determine if the packet is an SLAPprotocol packet. If it is, NS-vLB optionally proceeds to a block 207 andextracts an eNode-B IP address of an eNodeB from which or to which thepacket was sent. Based on the extracted eNodeB IP address, NS-vLBcomputes, optionally in a block 208, an address of a destination vProbe,to which to forward the packet. By way of example, to compute a vProbedestination address, in block 208 NS-vLB may hash the extracted IPaddress, and use a look-up table (LUT) that maps hashes of IP addressesof eNodeBs from which MME 31 may receive a packet to determine to whichvProbe, 121, 122, or 123 to forward the packet. In a block 209 NS-vLBmay encapsulate the packet for tunneling to the determined destinationvProbe address, and in a block 210 may transmit the packet to thedestination vProbe. Any suitable tunneling protocol such as the GRE(General Routing Encapsulation) tunneling protocol may be used in block210 to tunnel the packet to the destination vProbe. After transmittingthe S1AP packet to its destination vProbe, NS-vLB returns to block 202to receive another packet from the selected MME interfaces.

If a packet received by NS-vLB is found to be an L7 packet in decisionblock 204, but not an S1AP packet in decision block 206, NS-vLB mayproceed to determine in a decision block 212 if the received packet isan SGs protocol packet. If the packet is an SGs packet, optionally in ablock 213, NS-vLB extracts an MSC IP address from the packet.Optionally, in blocks 208, 209, 210, NS-vLB proceeds to determine an IPaddress of a destination vProbe 121, 122, or 123, to which to forwardthe packet based on the extracted MSC IP address, and tunnel the packetto the destination vProbe. Thereafter, NS-vLB may return to block 202 toreceive another packet.

If a received packet is determined to be an L7 packet but not an S1AP orSGs packet, NS-vLB may proceed from decision block 212 to a decisionblock 214 to determine if the packet is a GTP-C packet. If it is a GTP-Cpacket, NS-vLB may proceed to blocks 215 and 216 to tunnel the packet toGTP-C stateful load balancer S-vLB 110 (FIG. 2). Following transmissionof the packet to GTP-C S-vLB 110, NS-vLB may return to block 202 toreceive another packet. If on the other hand in decision block 214NS-vLB finds that the packet is not a GTP-C packet, NS-vLB may proceedto a block 217 to determine if the packet is a Diameter packet. If it isa Diameter packet, NS-vLB may proceed to blocks 218 and 219 to tunnelthe packet to Diameter stateful load balancer S-vLB 111 (FIG. 2), andthereafter return to block 202.

If on the other hand, in decision block 217 NS-vLB finds that the packetis not a Diameter packet, NS-vLB may proceed to blocks 220 and 221 totunnel the packet to a default vProbe or vProbe group, which may be anyone of vProbes 121, 122, or 123 and thereafter return to block 202.

FIGS. 4A-4B show a flow diagram of an algorithm 300 by which,optionally, Diameter stateful virtual load balancer S-vLB 111 (FIG. 2)processes Diameter packets it receives from non-stateful NS-vLB loadbalancers 101 and 102.

In a block 302 Diameter stateful load balancer S-vLB 111 receives anSCTP (Stream Control Message Protocol) packet comprising one or morecomplete and/or fragmented Diameter messages. In a decision block 304load balancer S-vLB 111 determines if the SCTP packet comprisesfragmented Diameter messages or a single, complete, non-fragmented,Diameter message. If the packet is a “fragmented packet”, comprisingfragmented Diameter messages, load balancer S-vLB 111 proceeds to ablock 306 and reassembles a Diameter message, a fragment of which iscomprised in the SCTP packet. Reassembly comprises aggregating asufficient number of SCTP packets to provide load balancer S-vLB 111with a sufficient number of fragments to enable reassembly of a completeDiameter message. Following reassembly, load balancer 111 may proceed toa decision block 308. If in decision block 304, load balancer S-vLB 111determines that the SCTP packet received in block 302 is not fragmented,load balancer S-vLB 111 may proceed directly to decision block 308.

In decision block 308 load balancer S-vLB 111 optionally determines ifthe Diameter message is an Authentication Information Request message,and if it is, may proceed to a block 310 in which the load balancerextracts an IMSI (International Mobile Subscriber Identity) and ahop-to-hop identifier (H2H ID), source and destination IP addresses thatrespectively identify a UE associated with the Diameter message and asession in which the UE is engaged. In a block 312 load balancer 111stores the IMSI, H2H ID, and source and destination addresses in atransaction table that the load balancer maintains in a memory.Thereafter the load balancer may proceed to a block 328 in which, basedon the extracted H2H ID it determines an IP address of a vProbe 121,122, or 123, to which the load balancer will forward the packet orpackets containing the Diameter message for processing. In a block 330load balancer S-vLB 111 encapsulates the Diameter message packet orpackets for tunneling to the determined IP address, and in a block 332transmits the Diameter message to the vProbe IP address.

In a block 334, load balancer S-vLB 111 determines if the transmittedDiameter message is a last Diameter message in the SCTP packet. If itis, load balancer S-vLB 111 proceeds to a junction “X” and returns toblock 302 to receive and process another SCTP packet. On the other hand,if it is not a last Diameter message, load balancer S-vLB 111 mayproceed to a junction “Y” and returns to block 304 to determine if thereis another Diameter message in the packet for processing and if it isfragmented.

In processing a Diameter message, if in decision block 308 load balancerS-vLB 111 determines that the Diameter message is not an AuthenticationInformation Request message, the load balancer may proceed to a decisionblock 314 to determine if the Diameter message is a response to anAuthentication Information Request message. If the message is not aresponse to an Authentication Information Request message load balancerS-vLB 111 may proceed from decision block 314 to a block 326 to extracta H2H ID from the message and in blocks 328-332 to transmit the messageto a vProbe having an IP address determined from the H2H ID. In block334 the load balancer determines whether to return to block 302 andreceive another SCTP packet or to block 304 to process another Diametermessage comprised in the current SCTP packet.

If in block 314 load balancer S-vLB 111 determines that the Diametermessage is an Authentication Information Answer message the loadbalancer may proceed to a block 316 and extract an H2H ID from themessage. In a block 318 the load balancer may attempt to match theextracted H2H ID, source and destination IP addresses with an H2H ID,source and destination IP addresses message previously logged into thetransaction table to identify a Diameter Authentication InformationRequest message with which the Authentication Information Answer messageis associated. In a decision block 320, if the matching attempt issuccessful, Diameter load balancer S-vLB 111 proceeds to a block 323 andgenerates a data record comprising the authentication vector extractedin block 316 from the Diameter Authentication Information Answer messageand the IMSI extracted from the associated Diameter AuthenticationInformation Request message. The combination of an encryption keycomprised in the authentication vector and the IMSI enable a vProbe 121,122, or 123 to decipher a packet that it receives which is associatedwith the session associated with the IMSI. In a block 324 load balancerS-vLB 111 may transmit the data record to each of vProbes 121, 122, and123.

Optionally, following transmission of the data record, load balancerS-vLB 111 may proceed to blocks 328-332 to determine a destination IPaddress of a vProbe 121, 122, or 123 to which the DiameterAuthentication Information Answer message should be sent. Thedestination IP address determined for the Authentication InformationAnswer message is based on the H2H ID associated with the AuthenticationInformation Answer message, which H2H ID is shared with theAuthentication Information Request message matched to the AuthenticationInformation Answer message in block 318. As a result, the vProbe IPaddress determined for the Diameter Authentication Information Answermessage is the same as the vProbe IP address determined for theassociated Diameter Authentication Information Request message. And inblock 332, after encapsulation in block 330, the Diameter AuthenticationInformation Answer message is transmitted to the same vProbe foranalysis to which the associated Diameter Authentication InformationRequest message was sent. After transmission of the DiameterAuthentication Information Answer, message load balancer 111 may proceedto block 334 to determine whether to return to block 302 and receiveanother SCTP packet or to block 304 to process the next Diameter messagein the current SCTP packet. It is noted that whereas in the abovedescription load balancer S-vLB 111 proceeds to blocks 328-332 afterhaving generated and transmitted an associated data record, in anembodiment load balancer S-vLB 111 may generate and transmit the datarecord after having proceeded to execute, or during execution of, blocks328-332.

In a case for which in decision block 320 load balancer S-vLB 111determines that matching in block 318 has failed, the load balancer mayproceed to perform a “Wait and See” procedure. In the Wait and Seeprocedure load balancer S-vLB 111 stores the Diameter AuthenticationInformation Answer message in a holding memory, optionally a RAM (randomaccess memory). In the holding memory the Diameter message may be storedwith other messages in order of their respective time stamps provided bythe non-Stateful load balancer NS-vLB 101 or NS-vLB 102 from which themessage was forwarded to load balancer S-vLB 111. In an embodiment, theload balancer S-vLB 111 processes the message in order of its time stampfollowing a predetermined from a time at which the message was stored inthe holding memory to make at least one additional attempt at matchingthe Authentication Information Answer message with an AuthenticationInformation Request message. If in a decision block 322 the at least oneadditional attempt is successful, load balancer S-vLB 111 may proceed toblocks 323 and 324 to generate and transmit an IMSI plus authenticationvector data record to all vProbes 121, 122, and 123. If however, indecision block 322 load balancer 111 determines that the at least oneadditional attempt has failed, the load balancer may proceed to executeblocks 328-334.

In an embodiment, to reduce a probability that an AuthenticationInformation Answer message reaches Diameter load balancer S-vLB 111before the associated Authentication Information Request message reachesthe load balancer, the load balancer may store all Diameter messagesthat it receives in a holding memory prior to attempting to matchmessages. The load balancer holds the messages in the holding memory fora given delay time before processing to reduce the probability that theload balancer will not find a match for an Authentication InformationAnswer

FIGS. 5A-5C show a flow diagram of an algorithm 400 by which,optionally, GTP-C, stateful virtual load balancer S-vLB 110 (FIG. 2)processes GTP-C packets it receives from non-stateful NS-vLB loadbalancers 101 and 102.

In a block 402 load balancer S-vLB 110 receives a GTP-C message packet,and optionally in a block 410 determines if the message is a createsession request. If it is, in a block 411 the load balancer extracts asequence number, an IMSI, and a GTP-C MME F-TEID (Fully Qualified TunnelEndpoint Identifier) located in the packet payload. In a block 412 loadbalancer S-vLB 110 may store the MME F-TEID and IMSI in a session tablethat the load balancer maintains, and in a block 413 stores the packetin a holding memory, optionally a RAM. From block 413 the load balancermay return to block 402 to receive another packet.

If in decision block 410 load balancer S-vLB 110 determines that themessage is not a create session message, the load balancer may proceedto a decision block 420 to determine if the message is a create sessionresponse message. If it is, load balancer S-vLB 110, optionally in ablock 421, extracts a sequence number and a GTP-C SGW F-TEID andoptionally proceeds to find a match for the extracted sequence numberwith a previously extracted create session request sequence numberstored in the session table. Load balancer S-vLB 110 may store the GTP-CSGW F-TEID in the session table and optionally proceed to a block 471.In block 471 based on the GTP-C MME F-TEID extracted from the associatedcreate session request and stored in the session table in block 412,load balancer 110 determines an IP address for a vProbe 121, 122, or 123to which to send for processing the session response packet and itsassociated session request packet stored in block 413 in RAM. In a block472, the load balancer encapsulates the packets for tunneling, and in ablock 473 transmits the encapsulated packets to the determinedIP-address of the destination vProbe. Load balancer S-vLB 110 may thenreturn to block 402 to receive another GTP-C packet.

If in decision block 420 load balancer S-vLB 110 determines that theGTP-C message is not a create session response, the load balancer mayproceed to a decision block 430 to determine if the message is a modifybearer request message. If it is, load balancer S-vLB 110 may proceed toa block 431 in which the load balancer extracts a GTP-C F-TEID an S1-UeNodeB F-TEID, and an eNodeB S1-U IP address from the packet. In a block432 load balancer 110 matches the extracted GTP-C TEID with a GTP-C SGWF-TEID previously stored in the load balancer session table for a createsession response and stores the S1-U eNodeB F-TEID and eNodeB S1-U IPaddress in the session table so that it may be associated with thecreate session response of the same GTP-C SGW F-TEID. In a block 434 theload balancer stores the packet in the holding memory RAM and may returnto block 402 to receive another packet.

If in decision block 430 load balancer S-vLB 110 determines that thepacket is not a modify bearer request, the load balancer may continue toa decision block 440 to determine if the message is a modify bearerresponse message. If it is a modify bearer response message, the loadbalancer may proceed to a block 441 to extract a GTP-C F-TEID from thepacket as well as an S1-U SGW F-TEID, and in a block 442 match theextracted GTP-C F-TEID with a GTP-C MME F-TEID previously stored in thesession table. In a block 443 load balancer S-vLB 110 stores theextracted S1-U SGW F-TEID in the session table so that it may beassociated with the other data in the session table having the sameGTP-C MME-TEID. In a block 444 load balancer S-vLB 110 generates a datarecord comprising the IMSI, S1-U eNodeB F-TEID, and the S1-U SGW F-TEIDthat are stored in the session table and associated with the same GTP-CMME F-TEID. In a block 445 the load balancer may determine a vProbe IPaddress based on the eNodeB S1-U IP address extracted in block 431 forthe modify bearer request associated with the modify bearer response.And in a block 446 the load balancer may transmit the data record to thevProbe 121, 122, or 123 having the determined vProbe IP address. Loadbalancer S-vLB 110 may then proceed to blocks 471-473 to transmit theassociated modify bearer request and modify bearer response packets forprocessing to a vProbe having an IP address determined from the GTP-CMME F-TEID that is associated with the modify session request andresponse packets. The vProbe to which the associated modify bearerrequest and modify bearer response packets are sent is also the samevProbe to which the create session request and response packetsassociated with the same GTP-C MME F-TEID were transmitted. From block473 the load balancer may return to block 402 to receive another packet.

In decision block 440 if load balancer S-vLB 110 determines that thepacket is not a modify bearer response, the load balancer may proceed toa block 450 to determine if the packet is another type of message packetdifferent from a create session or modify bearer request. If it is, in ablock 451 the load balancer extracts a GTP-C TEID from the packet and ina block 452 matches the extracted GTP-C TEID with a GTP-C MME TEID or aGTP-C SGW F-TEID previously stored in the session table. Load balancerS-vLB 110 may then proceed to blocks 471-473 to transmit the packet to avProbe having an IP address determined from the GTP-C MME F-TEIDassociated with the session to which the create session request andresponse packets associated with the same GTP-C MME F-TEID belong. Theload balancer may then proceed to block 402 to receive another packet.

It is noted that whereas a Wait and See procedure is described withrespect to load balancer S-vLB 111 it may also be used where necessaryby load balancer S-vLB 110.

The above discussion describes a Smart-vLB system balancing dataprocessing for packets transmitted via MME interfaces in accordance withan embodiment of the disclosure. However, a Smart-vLB in accordance withan embodiment is not limited to load balancing for packets transmittedvia MME interfaces, and may be configured to load balance forcommunication interfaces via which entities of a network other than anMME communicate. By way of example, FIG. 6 schematically shows aSmart-vLB 100 coupled to LTE network 20 operating to load balanceprocessing of data comprised in control plane and user plane packetstransmitted between the SGW node, the PGW node, and other nodes in theLTE network, in accordance with an embodiment of the disclosure. In thefigure Smart-vLB 100 is shown by way of example coupled to LTE 20 toreceive and load balance packets mirrored from interfaces S1-U, S4, S11,S5/S8, SGi, Gn, Gi, Gx. A Smart-vLB may simultaneously be coupled tointerfaces of more than one network entity and be configured tosubstantially simultaneously be responsible for load balancingcommunications for all the more than one network entity.

FIG. 7A shows a flow diagram of an algorithm 500 by which NS-vLB 101 orNS-vLB 102, may determine where to forward packets they receive from anSGW or PGW interface to which Smart-vLB 100 is coupled, in accordancewith an embodiment of the disclosure.

In a block 502 the NS-vLB (NS-vLB 101 or NS-vLB 102) receives andoptionally time stamps a packet from the selected SGW interfaces shownin FIG. 6, and in a decision block 504 determines if the packet containsa GTP-C protocol; packet. If it is a GTP-C packet, NS-vLB may proceed toblocks 506 and 508 to tunnel the packet to GTP-C stateful load balancerS-vLB 110 (FIG. 6). Following transmission of the packet to S-vLB 110,NS-vLB may return to block 502 to receive another packet. If on theother hand in decision block 504 NS-vLB finds that the packet is not aGTP-C packet, NS-vLB may proceed to a block 510 to determine if thepacket is a Diameter packet. If it is a Diameter packet, NS-vLB mayproceed to blocks 512 and 514 to tunnel the packet to Diameter statefulload balancer S-vLB 111 (FIG. 6), and thereafter return to block 502.

If on the other hand, in decision block 510 NS-vLB finds that the packetis not a Diameter packet, NS-vLB may proceed to decision block 516 todetermine if the packet is a GTP-U, Gi, or SGi packet and if not,discards the packet or transmits the packet to a default vProbe, andreturns to block 502 to receive another packet. On the other hand, if indecision block 510, NS-vLB determines that it is a GTP-U, Gu, or SGipacket, NS-vLB may proceed to a block 518. In block 518 NS-vLB mayextract an MS IP (Mobile Station IP) address from the packet and in ablock 520 the NS-vLB may determine an address for a destination vProbe,vProbe 121, 122, or 123, to which to transmit the packet for processing.In a block 522 NS-vLB encapsulates the packet for tunneling andtransmits the encapsulated packet to the determined address of thedestination vProbe. Following transmission NS-vLB may return to block502 to receive another packet.

FIG. 7B shows a flow diagram of an algorithm 600 by which, optionally,GTP-C, stateful virtual load balancer S-vLB 110 (FIG. 6) processes aGTP-C packet mirrored from a SGW interface to which Smart-vLB 100 iscoupled that S-vLB 110 receives from non-stateful NS-vLB load balancer101 or 102.

In a block 602 load balancer S-vLB 110 receives a GTP-C message packet,and optionally in a block 610 determines if the message is a createsession request. If it is, in a block 611 the load balancer extracts asequence number, an IMSI, and a GTP-C MME F-TEID (Fully Qualified TunnelEndpoint Identifier) located in the packet. In a block 612 load balancerS-vLB 110 may store the sequence number, MME F-TEID, and IMSI in asession table that the load balancer maintains, and in a block 613stores the packet in a holding memory, optionally a RAM. From block 613the load balancer may return to block 602 to receive another packet.

If in decision block 610 load balancer S-vLB 110 determines that themessage is not a create session message, the load balancer may proceedto a decision block 620 to determine if the message is a create sessionresponse message. If it is, GTP-C virtual load balancer S-vLB 110,optionally in a block 621, extracts a sequence number, a GTP-C SGWF-TEID and a MS IP address. Thereafter in a block 622, S-vLB 110optionally proceeds to find a match for the extracted sequence numberwith a previously extracted create session request sequence numberstored in the session table. In a block 623 load balancer S-vLB 110 maystore the GTP-C SGW F-TEID in the session table and, optionally, proceedto a block 650. In block 650 based on the MS IP address extracted fromthe associated create session request and stored in the session table inblock 623, load balancer S-vLB 110 determines an IP address for adestination vProbe 121, 122, or 123 to which to send for processing, thesession response packet and its associated session request packet storedin RAM in block 613. In a block 652, the load balancer may encapsulatethe packets for tunneling and transmit the encapsulated packets to thedetermined IP-address of the destination vProbe. Load balancer S-vLB 110may then return to block 602 to receive another GTP-C packet.

If in decision block 420 load balancer S-vLB 110 determines that theGTP-C message is not a create session response, the load balancer mayproceed to a decision block 630 to extract a GTP-C F-TEID from thepacket. In a block 632 load balancer S-vLB 110 matches the extractedGTP-C TEID with a GTP-C MME F-TEID or GTP-C SGW F-TEID previously storedin the load balancer session table and may proceed to block 650. Inblock 650 S-vLB 110 calculates an IP address for a destination vProbe121, 122, or 123 to which to send the packet for processing based on theMS IP address stored in the session table for the GTP-C MME F-TEID orGTP-C SGW F-TEID matched with the GTP-TEID in block 632. In a block 652,S-vLB 110 encapsulates the packet and transmits the packet to thedetermined IP address of the destination vProbe and may return to block602 to receive another packet.

In an embodiment Smart-vLB 100 may be configured as described below toload balance user packets in packet flows of EPS sessions between UEsand PDNs connected by LTE 20 and control, “throttle”, transmission ratesof copies of user packets to a probe for processing to preventoversubscribing probe capacity. PDNs to which an LTE may connect areidentified by PDN-IDs, which are conventionally also referred to asAccess Point Names (APNs), and EPS sessions between UEs and PDNs mayalso be referred to as PDN connections.

To perform the load balancing, Smart-vLB 100 may, as schematically shownin FIG. 6, be configured to receive and process control packets mirroredto Smart-vLB 100 from control plane interfaces, optionally, as discussedabove, as well as user packets from user plane interfaces, such asoptionally S1-U, S5/S8, and/or SGi interfaces. S1-U and S5/S8 interfacesprovide bearers that are concatenated with a data radio bearer (DRB) toprovide EPS bearers through which user packets in an EPS session aretunneled from an eNodeB in communication with LTE 20 to which a UE isconnected and a PDN to which LTE 20 connects the UE. Interface S1-Ugoverns user packet configuration and transportation through an S1bearer that provides the EPS bearers with a GTP-U tunnel to transportuser packets in the EPS session between the eNodeB to which the UE isconnected and a serving gateway SGW of the LTE. Interface S5/S8 governsuser packet configuration and transportation through an S5/S8 bearerwhich provides a GTP-U tunnel to transport user packets in the EPSsession between the SGW and a Packet Data Network Gateway (PGW) whichconnects LTE 20, and thereby the UE, to the PDN. The SGi interfaceconfigures communications between the PGW and the PDN.

In accordance with an embodiment, Smart-vLB 100 load balances andthrottles copies of user plane packets from UEs connected to LTE 20based on a particular feature, a target feature, that characterizes thepackets. As noted above, the target feature of the packets may by way ofexample, be an IMSI or an IMEI (International Mobile Equipment Identity)of a UE associated with transmission of the packets, an MSISDN (MobileStation International Subscriber Directory Number), APN (Access PointName), QCI, (Quality of Service Class Identifier), and/or a CGI (CellGlobal Identifier). For convenience of presentation, it is assumed thatthe target feature of the packets is their IMSI. FIG. 8 exhibits a flowdiagram 700 by which Smart-vLB 100 may associate user packets propagatedin EPS sessions through LTE 20 with IMSIs of the UEs participating inthe sessions, and based, optionally, on the IMSIs load balance the userpackets.

IMSIs of the UEs that the Smart-vLB associates with the user packets.FIG. 8 exhibits a flow diagram 700 by which Smart-vLB 100 may associateuser packets propagated in EPS sessions through LTE 20 with IMSIs of theUEs participating in the sessions, and based, optionally, on the IMSIsload balance the user packets.

In a block 702 Smart-vLB 100 may initiate at least one holding bufferfor each NS-vLB 101 and 102, in which the NS-vLB temporarily stores userplane packets it receives for which the NS-vLB cannot, or is not asdiscussed below, fully configured to associate an IMSI.

In a block 704 non-stateful NS-vLBs, by way of example NS-vLBs 101 and102, comprised in Smart-vLB 100 receive control and user plane packetsfrom interfaces that the Smart-vLB monitors for load balancing. In ablock 706 the NS-vLBs forward control plane packets for processing tostateful S-vLBs comprised in Smart-vLB 100, by way of example GTP-CS-vLB 110 and/or Diameter S-vLB 111. Optionally, in a block 708 theS-vLBs process control plane packets which they receive that areinvolved in establishing, modifying, tearing down, or configuring an EPSsession to acquire, in accordance with an embodiment of the disclosure,an ensemble, hereinafter also referred to as a characterizing vector, ofcharacterizing variables. A characterizing variable of an EPS session isa variable that may be used by itself or in conjunction with anothercharacterizing variable, to associate a user packet transported in theEPS session with an IMSI, which may also be referred to as a sessionIMSI, of a UE participating in the EPS session. A characterizingvariable may also be referred to as a component of a characterizingvector to which the characterizing variable belongs. The characterizingvariables may comprise various transport path variables, such as userand NE IDs, and/or user subscription variables. By way of example thecharacterizing variables may comprise, at least one, or any combinationof more than one, of a GUTI and/or IP address assigned by an MME in LTE20 to the UE for use in the EPS session, the APN of a PDN to which theLTE connects the UE for the session, at least one user plane TEID thatidentifies a tunnel end of a bearer comprised in the session EPS bearer,and/or an IP address of an NE in LTE 20 that supports the session.Characterizing variables that are associated with an IMSI of a sessionmay be determined by S-vLB 110 and/or S-vLB 110 111 similarly to themanner by which Smart-vLB 100 associates IMSIs with data items in dataand/or session records stored in a transaction or session tablemaintained by Smart-vLB 100 and described with reference to algorithms300, 400, and/or 500.

In an embodiment, the various S-vLBs that receive and process controlplane packets to identify components for a characterizing vector of asession may store the components that the S-vLB identifies in a sessionrecord in a session table maintained in a memory of the Smart-vLB. Whenthe session record comprises a number and variety of characterizingvariables sufficient to enable substantially any user packet propagatedvia LTE network 20 in the session to be associated with the sessionIMSI, the characterizing vector may be considered complete. Acharacterizing vector for an EPS session that connects a UE with a PDNmay be defined to have a predetermined number and type of components,such as by way of example components selected from the components notedabove, and when a value is assigned to each component, thecharacterizing vector may be considered to be complete. Optionally, acharacterizing vector is complete when Smart-vLB 100 associates a lastTEID that is used in transporting packets via the EPS session with thesession IMSI.

In a block 710 Smart-vLB 100 may determine that a characterizing vectorfor a given EPS session is complete and transmit to each NS-vLB 101, 102or a selection of NS-vLBs in the Smart-Vlb, the complete characterizingvector for use in associating user packets with the EPS session IMSI andload balancing the associated packets. In an embodiment, S-vLBs inSmart-vLB may be configured to communicate with each other and/or thesession table, to determine when a characterizing vector is complete. Toprovide each of the NS-vLBs with a complete characterizing vector theS-vLBs may operate to assign at least one of the S-vLBs to transmit acomplete characterizing vector to at least one of the NS-vLBs. In anembodiment, to provide Smart-vLB 100 with advantageously small latencyfor processing user packets, an S-vLB that operates to identifycharacterizing vector components may be configured to transmit thecomponents and/or the vector that it identifies to an NS-vLB using arelatively high-speed communications the high-speed protocol providestransit times for transmission of the components and/or the vectorbetween the S-vLB and NS-vLB that is less than 10 ms. Optionally, thecommunications protocol provides transit times less than about 5 ms.Optionally, the communications protocol provides transit times less thanabout 2 ms. In an embodiment the S-vLB high speed protocol is a Zer0MQprotocol.

In the above description, Smart-vLB 100 appears to be configured inaccordance with an embodiment to operate a centralized session table forcompilation and storage of characterizing vectors and to distribute acharacterizing vector to NS-vLBs only after values have been providedfor all of the vector components and the vector is considered to becomplete. However, practice of an embodiment of the disclosure is notlimited to transmitting complete characterizing vectors to NS-vLBs. Inan embodiment S-vLBs may be configured, to transmit characterizingcomponents for an EPS session, “piecemeal” substantially as they areidentified, to an NS-vLBs in Smart-vLB 100 that is configured to receivethe characterizing components piecemeal and compile the components intoa characterizing vector. For example, in response to identifying aparticular characterizing component of an EPS session, and optionallyindependent of identifying and/or transmitting another characterizingcomponent, an S-vLB may transmit a copy of the particular component toeach NS-vLB together with information sufficient for identifying an IMSIwith which the particular component is associated. The NS-vLB may storethe particular component it receives together with other components of acharacterizing vector associated with the same IMSI with which theparticular component is associated in a same record that the NS-vLBmaintains and uses to compile a characterizing vector.

Upon receiving and/or compiling a complete characterizing vector, anNS-vLB is fully configured to associate user packets in the given EPSsession with the session IMSI and to load balance the packets, andoptionally in a block 712, the NS-vLB begins using the characterizingvector to associate user packets in the NS-vLB buffer and user packetsas they are received by the NS-vLB, optionally without first storing thepackets in the buffer. Using the characterizing vector to associate auser packet with the given IMSI may comprise determining if a header ofthe user packet contains a data item that matches a component of thecharacterizing vector. For example, the NS-vLB may associate the userpacket with the given IMSI because the user packet header may comprise aTEID which the NS-vLB determines matches a component of thecharacterizing vector that is a TEID of an EPS bearer associated withthe given IMSI. Or the NS-vLB may find that the user packet header maycomprise an IP address and APN that matches components of thecharacterizing vector.

In a block 714 the NS-vLB may apply a selection filter to select fromamong user packets the NS-vLB has associated with an IMSI, a subset ofuser packets for load balancing and transmitting to v-Probes 121-123 forprocessing and to drop from transmitting to the v-Probes user packetsthat are not selected. In an embodiment, the NS-vLB forwards selecteduser packets associated with a same IMSI to a same vProbe 121-123 forprocessing. It is noted that in an embodiment the NS-vLB may operateproperly to forward selected user packets associated with a same IMSI toa same vProbe 121-123 for processing, independent of whether the UEassociated with the IMSI is assigned an IPv4, IPV6, or Dual Stack MS-IPaddress, and/or is assigned more than one MS-IP address.

In an embodiment of the disclosure Smart-vLB 100 comprises a library ofone or more selection filters that the NS-vLB may be configured to applyto select user packets for load balancing. Optionally, the selectionfilter library comprises a “depletion filter” that operates to controlor throttle a number of packets that Smart-vLB transmits to vProbes121-123 for processing. In an embodiment, the depletion filter operatesto select packets for load balancing and forwarding to vProbes 121-123,to limit a number of packets that Smart-vLB 100 transmits per unit time,also referred to as an LB transmission rate, to a given probe of vProbes121-123 or globally to all the vProbes for processing. Selection ofpackets to limit an LB transmission rate may be a function of at leastone LB transmission rate upper limit and an LB transmission rate ofSmart-vLB 100. For example, if the LB transmission rate of Smart-vLB 100is less than the at least one LB transmission rate upper limit, thedepletion filter may operate so that all user packets that the NS-vLBsassociate with an IMSI may be transmitted to a vProbe 121-123. On theother hand, the depletion filter may operate to select only a subset ofuser packets for transmission to vProbes 121-123 if in the absence ofselection, the LB transmission rate would exceed the transmission rateupper limit.

Optionally, the at least one LB transmission rate upper limit is basedon a capacity of at least one of vProbes 121-123 to process userpackets. In an embodiment, the upper limit is equal to a maximum packetprocessing rate of vProbes 121-123 or a fraction less than one of themaximum packet processing rate. The fraction less than one may, by wayof example be less than or equal to about 0.90. Optionally the fractionless than one is less than or equal to about 0.80. The upper limit maybe a function of time of day and/or a geographical region of LTE 20 forwhich the upper limit is to be applied. In an embodiment the upper limitis a dynamic limit continuously determined in accordance with aproportional-integral-derivative (PID) algorithm to maintain the loadbalancing rate at a predetermined desired rate.

In an embodiment, the depletion filter is configured to select userpackets received by NS-vLBs 121-123 for transmission based on the IMSIswith which the packets are associated so that only packets associatedwith an IMSI that is a member of a group of selected IMSIs are loadbalanced and forwarded to vProbes for processing. Optionally memberIMSIs are randomly selected.

Optionally, to implement selection of packets based on their respectiveIMSIs, Smart-vLB 100 may comprise at least one whitelist and at leastone blacklist of IMSIs. IMSIs listed in the at least one whitelist maybe selected for transmission to vProbes for processing and IMSIs listedin the at least one blacklist may be dropped. In an embodiment the atleast one whitelist and/or at least one blacklist may be dynamicallyupdated as a magnitude of packet traffic supported and/or demandedchanges. For example, a number of whitelisted IMSIs may be decreased asmagnitude of traffic increase and/or new IMSIs may be added or removedto the blacklist and/or whitelist as users connect to and/or disconnectfrom LTE 20.

In an embodiment a depletion filter, also referred to as a packet flowdepletion filter may operate to select user packets for forwarding tovProbes based on the respective packet flows to which the packetsbelong. Packets belonging to a same packet flow in LTE 20 are identifiedby a same 5-tuple of information elements that includes a source IP, adestination IP, a source port number, a destination port number, and acommunications protocol. A packet flow depletion filter that NS-vLBs inSmart-vLB 100 may apply to packets that Smart-vLB 100 receives from LTE20 operates to select packets for load balancing from a portion of thepacket flows in LTE 20 from which Smart-vLB 100 receives packets. TheNS-vLBs forward packets selected by the packet flow depletion filterthat belong to a same packet flow to a same vProbe.

In an embodiment the packet flow depletion filter is configured tooperate based on a set of operating parameters optionally comprising anactivation packet traffic threshold, activation delay, deactivationpacket traffic threshold, deactivation delay, and dynamic depletionfactor. The packet flow depletion filter is configured to activate todrop packets belonging to a fraction of the packet flows extant in LTE20 following an activation delay from a time at which packet trafficthat Smart-vLB 100 receives from LTE 20 exceeds the activation packettraffic threshold. The fraction of the packet flows from which theNS-vLBs drop packets may be substantially equal to the dynamic depletionfactor. The NS-vLBs determine the dynamic depletion factor as a functionof magnitude of packet traffic that Smart-vLB 100 receives from LTE 20so that a transmission rate of packets that Smart-vLB 100 forwards tovProbes 121-123 for processing does not exceed capacity, that isoptionally a maximum processing rate, of the vProbes to process thepackets. The packet flow depletion filter is configured to deactivateand cease dropping packets following a deactivation delay from a time atwhich packet traffic that Smart-vLB 100 receives from LTE 20 drops belowthe deactivation activation packet traffic threshold. Optionally, theactivation and deactivation packet traffic thresholds are substantiallythe same. In an embodiment the activation and deactivation packettraffic thresholds are different. Optionally, the activation anddeactivation delays are substantially the same. In an embodiment theactivation and deactivation delays are different.

It is noted that Smart-vLB 100 may be configured to apply a packet flowdepletion filter to throttle packet transmission to vProbes 121-123without having recourse to associating packets with an IMSI or aparameter other than an IMSI.

In an embodiment, the filter library comprises a “discrimination filter”that may be applied to select a portion of packets that Smart-vLB 100receives for load balancing based on features of user subscriberagreements under which users access and transmits packets over LTE 20. Adiscrimination filter may be operable to select packets for transmissionand processing by vProbes 121-123 based on features such as QoS, dataservices, and/or roaming policies of subscriber agreements associatedwith IMSIs. By way of example, NS-vLBs 101 and 102 may select packets asa function of QoS associated with an IMSI subscriber policy so that onlypackets associated with a QoS greater than a threshold QoS are selectedfor load balancing and forwarding to vProbes 121-123.

In an embodiment Smart-vLB 100 may use more than one selection filter toselect packets that the Smart-vLB transmits to a vProbe 121-123. Forexample, Smart-vLB 100 may use a depletion filter together with adiscrimination filter to select and limit a number of packets thatSmart-vLB load balances and forwards to vProbes 121-123. Thediscrimination filter may be used to select the packets based on a QoSthreshold determined responsive to a LB transmission rate upper limitdetermined by a depletion filter. In an embodiment the QoS threshold mayrise as a number of user packets that Smart-vLB 100 receives from LTE 20for load balancing increases.

It is noted that whereas in the above description packets were selectedfor load balancing and/or throttling based on their association with anIMSI, packets may be selected for load balancing and/or throttling inaccordance with an embodiment of the disclosure based on a parameter orparameters other than an IMSI. For example, packets may be selectedand/or throttled based on an IMEI, an MSISDN, GUTI, APN, QCI, and/or aCGI with which they may be associated. It is also noted that whereasSmart-vLB 100 is schematically shown and described as load balancing fora 4G LTE communications network a Smart-vLB in accordance with anembodiment may operate to load balance and throttle for 5G architectureand other networks configured to implement control and user planeseparation (CUPS). A Smart-vLB in accordance with an embodiment may beadvantageous for load balancing CUPS configured communications networksbecause the Smart-vLB architecture may use entities that operate asstateful load balancers which are different from entities that operateas non-stateful load balancers. For example, the functionalities ofNS-vLBs in Smart-vLB 100 may be provided by entities that are differentfrom entities that provide the functionalities of S-vLBs 110-111. As aresult, Smart-vLB 100 may provide advantageous processing efficienciesand scaling flexibilities by coupling NS-vLBs 101-102 to receive andprocess user plane packets in a CUPS configured communications networkand S-vLBs 119-111 to receive and process control plane packets from theCUPS network.

It is also noted that whereas Smart-vLB 100 is described as initiatingload balancing and/or throttling only after receiving a completecharacterizing vector, in an embodiment a Smart-vLB may operate to loadbalance using characterizing components of a characterizing vector asthe Smart-vLB receives them, without waiting to receive all thecharacterizing components of the characterizing vector.

There is therefore provided in accordance with an embodiment of thedisclosure, a load balancing system for a packet switched communicationsnetwork, the load balancing system comprising: at least one statefulload balancer configured to receive and process copies of control planepackets propagated by the network to determine a characterizing vectorfor each target feature of at least one target feature thatcharacterizes packets propagated by the network, which vector comprisesone or more characterizing components useable to associate packetspropagated by the network with the target feature; and at least onenon-stateful load balancer configured to: receive the at least onecharacterizing vector and copies of packets propagated by the network;use the at least one characterizing vector to associate a target featureof the at least one target feature with each of a plurality of thereceived packets; and load balance and transmit copies of packets thatthe non-stateful load balancer associates with a target feature of theat least one target feature to at least one probe, and copies of packetsassociated with a same target feature to a same probe of the at leastone probe.

Optionally, the at least one non-stateful load balancer is configured tobegin using components of the characterizing vector to associate copiesof packets with a given target feature of the at least one targetfeature only after receiving all the components of the characterizingvector for the given target feature. Optionally the at least onenon-stateful load balancer comprises a holding buffer and is configuredto store in the buffer packets associated with the given target featurethat it receives prior to using the characterizing components.

In an embodiment, the at least one non-stateful load balancer isconfigured to receive piecemeal the one or more characterizingcomponents for a given target feature of the at least one target featureand to begin using the characterizing components to associate packetswith the given target feature substantially at times at which thecomponents are received.

In an embodiment, the at least one stateful load balancer is configuredto transmit the one or more components of a characterizing vector to theat least one non-stateful load balancer using a high-speedcommunications protocol. Optionally, the high-speed protocol providestransit times for transmission of the one or more components from the atleast one stateful load balancer to the at least one non-stateful loadbalancer that is less than about 10 ms (milliseconds), less than about 5ms, or less than about 2 ms.

In an embodiment, the at least one non-stateful load balancer isconfigured to process copies of packets propagated in the communicationsnetwork to determine a characterizing vector for a target feature of theat least one target feature.

In an embodiment, the load balancing system comprises a library havingone or more selection filters that the at least one non-stateful loadbalancer may be configured to apply to select from among packets thatthe at least one non-stateful load balancer associates with a targetfeature of the at least one target feature, a subset of packets for loadbalancing and/or transmitting to the at least one probe for processing.Optionally, the one or more selection filters comprises a depletionfilter operable to select a portion of the received packets for loadbalancing and transmitting to the at least one probe and to drop packetsthat are not selected to limit a transmission rate to the at least oneprobe. Optionally, the depletion filter is configured to select a packetfor load balancing and transmitting based on a function of at least onetransmission rate upper limit and a transmission rate of the at leastone non-stateful load balancer. The at least one transmission rate upperlimit may be based on a capacity of the at least one probe to processpackets that the at least one probe receives from the at least onenon-stateful load balancer. The at least one transmission rate upperlimit may be a fraction less than one of a maximum packet processingrate of the at least one probe.

In an embodiment, the at least one transmission rate upper limit is adynamic limit configured to repeatedly be updated. Optionally the upperlimit is updated in accordance with a proportional-integral-derivative(PID) algorithm.

In an embodiment, the depletion filter operates to drop packets if inthe absence of dropping packets, the transmission rate would exceed theat least one transmission rate upper limit.

In an embodiment, the one or more selection filters comprises adepletion filter that has an activation packet traffic threshold andactivation delay and is configured to activate to drop packets at agiven time if packet traffic in the communications network is in excessof the activation packet traffic threshold for a period of time endingat the given time that is greater than the activation delay. Optionally,the depletion filter having the activation packet traffic thresholdcomprises a deactivation packet traffic threshold and deactivation delayand is configured to deactivate and cease dropping packets at a giventime after activation if packet traffic that the at least onenon-stateful load balancer receives is less than the deactivation packettraffic threshold for a period of time ending at the given time that isgreater than the deactivation delay.

In an embodiment, the one or more selection filters comprises adiscrimination filter that operates to select received packets for loadbalancing and/or transmitting to the at least one probe based onfeatures of user subscriber agreements under which users access andtransmits packets over the communications network.

In an embodiment, the at least one target feature comprises at least oneand any combination of more than one of an IMSI (International MobileSubscriber Identity), an IMEI (International Mobile Equipment Identity),an MSISDN (Mobile Station International Subscriber Directory Number),APN (Access Point Name), QCI, (Quality of Service Class Identifier), aCGI (Cell Global Identifier), and/or a flow in the communicationsnetwork to which a packet belongs.

In an embodiment, copies of packets that the non-stateful load balancersreceive are copies of user plane packets.

In an embodiment, copies of packets that the non-stateful load balancersreceive are copies of control plane packets.

In the description and claims of the present application, each of theverbs, “comprise” “include” and “have”, and conjugates thereof, are usedto indicate that the object or objects of the verb are not necessarily acomplete listing of components, elements or parts of the subject orsubjects of the verb.

Descriptions of embodiments of the disclosure in the present applicationare provided by way of example and are not intended to limit the scopeof the disclosure. The described embodiments comprise differentfeatures, not all of which are required in all embodiments of thedisclosure. Some embodiments utilize only some of the features orpossible combinations of the features. Variations of embodiments of thedisclosure that are described, and embodiments of the disclosurecomprising different combinations of features noted in the describedembodiments, will occur to persons of the art. The scope of thedisclosure is limited only by the claims.

The invention claimed is:
 1. A load balancing system for a packetswitched communications network, the load balancing system comprising: aplurality of probes for processing data from the network; at least onestateful load balancer configured to receive control plane packetsmirrored from interfaces in the network and process the received packetsdetermine for each target feature of a plurality of target features thatcharacterize packets propagated by the network, a characterizing vectorhaving one or more components associated with the target feature; and atleast one non-stateful load balancer configured to: receive thedetermined characterizing vector for each target feature from the atleast one stateful load balancer; receive packets mirrored frominterfaces in the network; associate each of a plurality of the packetsmirrored to the at least one non-stateful load balancer with a targetfeature of the plurality of target features using the receivedcharacterizing vector determined for the target feature; and loadbalance and transmit packets that the non-stateful load balancerassociates with the same target feature to a same at least one probe ofthe plurality of probes for processing.
 2. The load balancing systemaccording to claim 1 wherein the at least one non-stateful load balanceris configured to begin using the characterizing vector for a giventarget feature of the plurality of target features only after receivingall the components of the characterizing vector.
 3. The load balancingsystem according to claim 2 wherein the at least one non-stateful loadbalancer comprises a holding buffer and is configured to store in thebuffer packets associated with the given target feature that it receivesprior to using the characterizing vector.
 4. The load balancing systemaccording to claim 1 wherein the at least one non-stateful load balanceris configured to receive piecemeal the one or more components of thecharacterizing vector for a given target feature of the plurality oftarget features and to begin using the characterizing vector toassociate packets with the given target feature substantially at timesat which the components are received.
 5. The load balancing systemaccording to claim 1 wherein the at least one stateful load balancer isconfigured to transmit the one or more components of the characterizingvector for a given target feature of the plurality of target features tothe at least one non-stateful load balancer using a high-speedcommunications protocol.
 6. The load balancing system according to claim5 wherein the high-speed protocol provides transit times fortransmission of the one or more components from the at least onestateful load balancer to the at least one non-stateful load balancerthat is less than about 10 ms (milliseconds), less than about 5 ms, orless than about 2 ms.
 7. The load balancing system according to claim 1wherein the at least one non-stateful load balancer is configured toprocess received packets mirrored from an interface in thecommunications network to determine the characterizing vector for atarget feature of the at least one target feature.
 8. The load balancingsystem according to claim 1 and comprising a library having one or moreselection filters that the at least one non-stateful load balancer mayapply to select from among packets that the at least one non-statefulload balancer associates with a target feature of the plurality oftarget features, a subset of packets for load balancing and/ortransmitting to the at least one probe for processing.
 9. The loadbalancing system according to claim 8 wherein the one or more selectionfilters comprises a depletion filter operable to select a portion of thepackets mirrored to the load balancing system for load balancing andtransmitting to the at least one probe, and to drop packets that are notselected to limit a transmission rate to the at least one probe.
 10. Theload balancing system according to claim 9 wherein the depletion filteris configured to select a packet for load balancing and transmittingbased on a function of at least one transmission rate upper limit and/ora transmission rate of the at least one non-stateful load balancer. 11.The load balancing system according to claim 10 wherein the at least onetransmission rate upper limit is based on a capacity of the at least oneprobe to process packets that the at least one probe receives from theat least one non-stateful load balancer.
 12. The load balancing systemaccording to claim 11 wherein the at least one transmission rate upperlimit is a fraction less than one of a maximum packet processing rate ofthe at least one probe.
 13. The load balancing system according to claim10 wherein the at least one transmission rate upper limit is a dynamiclimit configured to repeatedly be updated.
 14. The load balancing systemaccording to claim 13 wherein the upper limit is updated in accordancewith a proportional-integral-derivative (PID) algorithm.
 15. The loadbalancing system according to claim 10 wherein the depletion filteroperates to drop packets if in the absence of dropping packets, thetransmission rate would exceed the at least one transmission rate upperlimit.
 16. The load balancing system according to claim 8 wherein theone or more selection filters comprises a depletion filter that has anactivation packet traffic threshold and activation delay and isconfigured to activate to drop packets at a given time if packet trafficin the communications network is in excess of the activation packettraffic threshold for a period of time ending at the given time that isgreater than the activation delay.
 17. The load balancing systemaccording to claim 16 wherein the depletion filter having the activationpacket traffic threshold comprises a deactivation packet trafficthreshold and deactivation delay and is configured to deactivate andcease dropping packets at a given time after activation if packettraffic that the at least one non-stateful load balancer receives isless than the deactivation packet traffic threshold for a period of timeending at the given time that is greater than the deactivation delay.18. The load balancing system according to claim 8 wherein the one ormore selection filters comprises a discrimination filter that operatesto select received packets for load balancing and/or transmitting to theat least one probe based on features of user subscriber agreements underwhich users access and transmits packets over the communicationsnetwork.
 19. The load balancing system according to claim 1 wherein theplurality of target feature comprises at least one or any combination ofmore than one of an IMSI (International Mobile Subscriber Identity), anIMEI (International Mobile Equipment Identity), an MSISDN (MobileStation International Subscriber Directory Number), APN (Access PointName), QCI, (Quality of Service Class Identifier), a CGI (Cell GlobalIdentifier), and/or a flow in the communications network to which apacket belongs.
 20. The load balancing system according to claim 1wherein the mirrored packets that the non-stateful load balancersreceive comprise user plane packets.
 21. The load balancing systemaccording to claim 1 wherein the mirrored packets that the non-statefulload balancers receive comprise control plane packets.